Improved Preconditioner for Hessian Free Optimization
نویسنده
چکیده
We investigate the use of Hessian Free optimization for learning deep autoencoders. One of the critical components in that algorithm is the choice of the preconditioner. We argue in this paper that the Jacobi preconditioner leads to faster optimization and we show how it can be accurately and efficiently estimated using a randomized algorithm.
منابع مشابه
Preconditioning for Hessian-Free Optimization
Recently Martens adapted the Hessian-free optimization method for the training of deep neural networks. One key aspect of this approach is that the Hessian is never computed explicitly, instead the Conjugate Gradient(CG) Algorithm is used to compute the new search direction by applying only matrix-vector products of the Hessian with arbitrary vectors. This can be done efficiently using a varian...
متن کاملAutomatic Preconditioning by Limited Memory Quasi-Newton Updating
This paper proposes a preconditioner for the conjugate gradient method (CG) that is designed for solving systems of equations Ax = bi with different right-hand-side vectors or for solving a sequence of slowly varying systems Akx = bk. The preconditioner has the form of a limited memory quasi-Newton matrix and is generated using information from the CG iteration. The automatic preconditioner doe...
متن کاملPreconditioning the Pressure Tracking in Fluid Dynamics by Shape Hessian Information
Potential flow pressure matching is a classical inverse design aerodynamic problem. The resulting loss of regularity during the optimization poses challenges for shape optimization with normal perturbation of the surface mesh nodes. Smoothness is not enforced by the parameterization but by a proper choice of the scalar product based on the shape Hessian, which is derived in local coordinates fo...
متن کاملAutomatic Preconditioning by Limited Memory
The paper proposes a preconditioner for the conjugate gradient method CG that is designed for solving systems of equations Ax bi with di erent right hand side vec tors or for solving a sequence of slowly varying systems Akx bk The preconditioner has the form of a limited memory quasi Newton matrix and is generated using infor mation from the CG iteration The automatic preconditioner does not re...
متن کاملImpulse Response Approximations of Discrete Shape Hessians with Application in CFD
Abstract. This paper discusses the symbol of the Hessian of a shape optimization problem in a viscid, incompressible flow. The symbol of the Hessian for the Stokes shape optimization problem is analytically approximated by tracking a Fourier mode through the operators involved. We propose a discrete method for finding the symbol and confirm the correctness by comparison with the analytical data...
متن کامل